Skip to content

fix: bound native scan claim queue reads - #3662

Merged
Patrick-Erichsen merged 1 commit into
codex/clawscan-claim-recoveryfrom
codex/clawscan-claim-read-budget
Sep 14, 2026
Merged

Patrick-Erichsen merged 1 commit into
codex/clawscan-claim-recoveryfrom
codex/clawscan-claim-read-budget

Conversation

@Patrick-Erichsen

@Patrick-Erichsen Patrick-Erichsen commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Native ClawScan workers requesting one job currently fetch up to 512 ready queue rows when generic GitHub rollout is disabled. Start each source page at the worker's remaining capacity, expanding to the existing bounded window only after encountering blocked legacy GitHub jobs. Preserve priority ordering, rollout restrictions, catalog admission and lease behavior.

Before/after proof on a disposable local Convex backend (http://127.0.0.1:3318) with 600 queued native bulk-rescan jobs:

Claim size Queue rows before → after (regression test) Real Convex document budget Before → after
1 512 → 1 14 Read-limit error → 1 claimed
4 512 → 4 26 Read-limit error → 4 claimed
16 512 → 16 74 Read-limit error → 16 claimed

The existing 513-blocked-GitHub-job fallback still reaches the allowed NVIDIA job in at most three source pages. All 117 security-scan tests pass after rebasing onto current main. Temporary seed/proof functions were deployed only locally, excluded from this PR, and the disposable backend was stopped.

This reduces demonstrated read amplification; it is not yet a demonstrated throughput fix. Local burst tests with 128 one-job requests at concurrency 16 and 32 still encountered heavy OCC contention before and after, with no consistent throughput improvement. The production claim-error cause remains unverified. Under renewed September 14 authorization, deploy this bounded read fix and measure actual claim failures and successful scan throughput before qualifying higher capacity. Scans continue at reduced concurrency during repair; raw claim errors still count toward backoff thresholds.

Validation:

  • bunx vitest run convex/securityScan.test.ts — 117 passed; new regression cases fail on the original code.
  • bun run ci:unit — 6,677 passed, 3 skipped; coverage gate passed.
  • bun run ci:static — passed.
  • bun run ci:types-build — passed.
  • bunx tsc -p convex/tsconfig.json --noEmit — passed.
  • Repository autoreview — no actionable findings.

@clawsweeper

clawsweeper Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@vercel

vercel Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
clawhub Ready Ready Preview Sep 14, 2026 9:46pm UTC

Request Review

@clawsweeper

clawsweeper Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed September 14, 2026, 5:47 PM ET / 21:47 UTC (Revision 4).

ClawSweeper review

What this changes

The PR sizes scan-queue pages to remaining worker capacity, expands them when blocked legacy jobs require it, and adds regression tests and a documented invariant.

Merge readiness

Ready for maintainer review

The fix remains useful: current main still performs the oversized reads. No blocking patch defects were found, and the supplied local Convex measurements support the narrowly stated read-budget improvement.

Priority: P2
Reviewed head: 4b956759b4681b73bee409340ff30db5ecf33383

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused repair with relevant regression coverage and concrete real-backend measurements, with its performance claims appropriately bounded.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The supplied local Convex measurements exercise the queue-claim mutation with 600 native jobs and show successful 1-, 4-, and 16-job claims under budgets that previously failed; they establish reduced read amplification, not improved production throughput.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The supplied local Convex measurements exercise the queue-claim mutation with 600 native jobs and show successful 1-, 4-, and 16-job claims under budgets that previously failed; they establish reduced read amplification, not improved production throughput.
Evidence reviewed 7 items Introduced scope: The pinned base-to-head change contains only queue page sizing, regression coverage, and the security-moderation invariant: 45 additions and 4 deletions across three files.
Current main still needs the fix: Current main selects MAX_CODEX_SCAN_CLAIM_LIMIT when GitHub skill synchronization is disabled, irrespective of remaining worker capacity. No implementation-on-main conclusion is supported; inspection of the supplied release tag's source was unavailable.
Production boundary and preserved checks: Both worker claim actions authenticate before calling the shared mutation. The mutation retains indexed source ordering, legacy rollout checks, catalog admission, final eligibility rechecks, and lease creation. The changed pagination uses Convex directly, establishing applicability of the repository's Convex guidance.
Findings None None.
Security None None.

How this fits together

ClawHub’s scan queue assigns pending skill and package security scans to authenticated workers. Queue selection applies source priorities and rollout restrictions before issuing leases for artifact inspection.

flowchart TD
  A[Worker requests capacity] --> B[Authenticate worker]
  B --> C[Read ready queue pages]
  C --> D[Check rollout and catalog eligibility]
  D -->|Blocked legacy jobs| C
  D -->|Eligible jobs| E[Issue scan leases]
  E --> F[Inspect skill or package artifacts]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Change size +45/-4 across 3 files The introduced patch stays confined to one queue-selection behavior, its tests, and its specification.
Production versus test LOC Production +9/-4, tests +32/-0, specification +4/-0 The five-line net production increase implements adaptive pagination with a stated read-amplification justification.

Technical review

Best possible solution:

Keep capacity-sized native reads and the existing legacy fallback, while qualifying any later concurrency increase through measured production errors and successful scan throughput.

Do we have a high-confidence way to reproduce the issue?

Yes: current main explicitly requests 512 queue entries for small native claims when rollout is disabled, and the supplied local Convex results exercise that case. This review verified the source path without executing it.

Is this the best way to solve the issue?

Yes, for the read-amplification defect: changing the shared pagination owner is narrower than altering worker capacity or introducing new indexes, while retaining the broad legacy fallback avoids excessive small queries.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 9be7637941d8.

Labels

Label justifications:

  • P2: This is a bounded scanner queue efficiency repair with demonstrated read-budget benefits but no established urgent production throughput fix.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The supplied local Convex measurements exercise the queue-claim mutation with 600 native jobs and show successful 1-, 4-, and 16-job claims under budgets that previously failed; they establish reduced read amplification, not improved production throughput.
  • proof: sufficient: Contributor real behavior proof is sufficient. The supplied local Convex measurements exercise the queue-claim mutation with 600 native jobs and show successful 1-, 4-, and 16-job claims under budgets that previously failed; they establish reduced read amplification, not improved production throughput.

Evidence

What I checked:

  • Introduced scope: The pinned base-to-head change contains only queue page sizing, regression coverage, and the security-moderation invariant: 45 additions and 4 deletions across three files. (convex/securityScan.ts:3201, 4b956759b468)
  • Current main still needs the fix: Current main selects MAX_CODEX_SCAN_CLAIM_LIMIT when GitHub skill synchronization is disabled, irrespective of remaining worker capacity. No implementation-on-main conclusion is supported; inspection of the supplied release tag's source was unavailable. (convex/securityScan.ts, 9be7637941d8)
  • Production boundary and preserved checks: Both worker claim actions authenticate before calling the shared mutation. The mutation retains indexed source ordering, legacy rollout checks, catalog admission, final eligibility rechecks, and lease creation. The changed pagination uses Convex directly, establishing applicability of the repository's Convex guidance. (convex/securityScan.ts:3116, 4b956759b468)
  • Focused regression coverage: New cases assert 1, 4, and 16 returned queue rows for matching native claim sizes. The legacy fixture still reaches an allowed NVIDIA job behind 513 blocked jobs within three source pages. These mocked tests supplement the reported real-backend measurements. (convex/securityScan.test.ts:4263, 4b956759b468)
  • Real Convex observations: The supplied complete PR body reports a disposable local Convex backend with 600 native jobs: document budgets of 14, 26, and 74 failed before the fix and allowed 1, 4, and 16 claims afterward. It explicitly reports continuing OCC contention and no consistent throughput improvement. Captured context identity: 82db1bc2d9e9e7efc9c3cd489a608f0e33866fb57c29c1b8a920837a8fd0f069. No execution was performed during this review.
  • Related work is distinct: fix: back off transient security worker claim errors #3695 remains open and supplies this PR's exact base commit. It addresses transient worker claim retries, whereas this PR reduces backend queue reads; neither replaces the other. (acecd5bc307e)

Likely related people:

  • Patrick-Erichsen: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (3 earlier review cycles)
  • reviewed 2026-09-11T00:51:37.327Z sha 18ddd2c :: blocked before merge. :: none
  • reviewed 2026-09-14T21:15:00.655Z sha 6c6bbf2 :: blocked before merge. :: none
  • reviewed 2026-09-14T21:23:16.970Z sha 2c7c10c :: needs maintainer review before merge. :: none

@blacksmith-sh

This comment has been minimized.

@Patrick-Erichsen
Patrick-Erichsen force-pushed the codex/clawscan-claim-read-budget branch from 18ddd2c to 6c6bbf2 Compare September 14, 2026 21:10
@Patrick-Erichsen
Patrick-Erichsen marked this pull request as ready for review September 14, 2026 21:10
@Patrick-Erichsen
Patrick-Erichsen force-pushed the codex/clawscan-claim-read-budget branch from 6c6bbf2 to 2c7c10c Compare September 14, 2026 21:18
@Patrick-Erichsen
Patrick-Erichsen changed the base branch from main to codex/clawscan-claim-recovery September 14, 2026 21:18
@Patrick-Erichsen
Patrick-Erichsen added this pull request to stack #3696 September 14, 2026 21:18
@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Sep 14, 2026
@Patrick-Erichsen
Patrick-Erichsen force-pushed the codex/clawscan-claim-read-budget branch from 2c7c10c to 4b95675 Compare September 14, 2026 21:43
@Patrick-Erichsen
Patrick-Erichsen merged commit 595afc0 into main Sep 14, 2026
35 of 36 checks passed
@Patrick-Erichsen
Patrick-Erichsen deleted the codex/clawscan-claim-read-budget branch September 14, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal backlog priority with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant